Next-Gen Characters: From 
Facial Scans to Facial Animation 

John Hable 
FilmicWorld.com 
@FilmicWorlds 



Facial Scans 


• Raw scans look great 

• How do we rig and animate it 
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Rig 

• Problem: Define a rig that matches these deformations 

• Just geometry? 

• No: Need diffuse animation too 







Diffuse Textures: 
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Diffuse Textures: 



point 4 


point 6 


point 2 





Diffuse Textures: 

• Diffuse map captures detail from skin stretching 

• It's not "just wrinkles" 

• Bloodflow is movement too 

• Everyone focuses on SPATIAL resolution 

• We need more TEMPORAL resolution 


How do we create a rig that: 

1. Maintains the geometry of the scans 

2. Maintains the diffuse color of the scans 

3. Is driven by standard mocap 


Background: 

1. Georgia Tech (did something) 

2. EAWW Vis Group 

• Tiger Woods 2007 (Game and E3 Demo) 

• NFS: Carbon 

3. LMNO (Spielberg title at EA) 

4. Naughty Dog 

5. Contract Work (Microsoft) 


Playable Ucap 

• GPU Gems 3: Playable Universal Capture 

• Tiger Woods 2007 and NFS: Carbon 

• Similar to LA Noire 

• Geometry: Nothing fancy 

• Diffuse Map: Animates each frame 

• Effectively playing a movie on their face 


Playable Ucap 

• Best facial animation at the time 

• Joint Geometry: Deep in uncanny valley 

• Joint Geometry + Animated Diffuse: 
Looks amazing. 

• Conclusion: We need animated Diffuse 

• Never showed the best data 


PTMs: 


• Facial Performance Synthesis using Deformation-Driven Polynomial 
Displacement Maps 

• http://Rl.ict.usc.edu/Research/PDM/ 


PTMs: 

• Facial Performance Synthesis using Deform 
Displacement Maps 

• http://gl.ict.usc.edu/Research/PDM/ 


happy expression, texture space 


-Driven Polynomial 
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PTMs: 

• Facial Performance Synthesis using Deformation-Driven Polynomial 
Displacement Maps 

• http://gl.ict.usc.edu/Research/PDM/ 



synthesized high-res geometry 


ground-truth high res geometry 


Very Interesting Paper: 

• Ucap: 

• Large quantity of high quality data 

• PDM Paper: 

• Small quantity of high quality data 

• 6 expressions, about a second each 

• Large quantity of low quality data 

• Use low quality data to drive high quality data 


Takeaway: Goal should be... 

• Short volume of high quality data 

• Captures subtle face details 

• Large quantity of low quality data 

• Markers 


Recent Advances: 


• Leveraging Motion Capture and 3D Scanning for High-fidelity Facial 
Performance Acquisition (Haoda Huang, Xin Tong, Hsaing-Tao Wu) 

• Capture may FACE poses 

• Replaces short 4D capture 

• facultv.cs.tamu.edu/ichai/proiects/face-TQG-2011/Face-final-vll.pdf 



Figure 1: Our system captures high-fidelity facial performances with realistic dynamic wrinkles and fine-scale facial details. 



Recent Advances: 

• Vuvuzela (used on Digital Ira) 

• Alignment of scans using tracking and optical flow 

• http://www. youtube. com/watch?v=lstcFOGwvU4 

• Phenomenal alignment 



artist remesh scan texture 


Recent Advances: 


• Photogrammetry makes massive improvements (Agisoft Photoscan) 

• Infinite-Realities 
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Workflow Overview: 

0) Scan actor. 

1) Process raw scans into aligned blendshapes. 

2) Compress blendshape textures for realtime playback. 

3) Drive blendshapes with mocap 

4) Render with skin shading 


Step 0: Scan 


Solution Used: Photogram metry 

• Infinite Realities with Lee Perry-Smith 

• Facial setup with 48 cameras 

• Sync with light 

• Also has 150 camera setup for full body 

• (and growing) 

• You've probably seen him before... 




Photogrammetry Pros: 

• Primary Advantage: Instant Capture 

• Sync with light (flash) 

• No alignment fixup 

• Can capture fine wrinkles (but not pores) 

• Only manual cleanup is ears/mouth 





Photogrammetry Cons: 

• No Fine Normals 

• You will have to sculpt or extract your normals from the diffuse 

• Some lighting baked in 

• Fresnel is an issue 


Infinite Reality Scans: 

• In this case, 70 expressions with highest quality solve in Photoscan 
• 7.5m polys 




Infinite Reality Scans: 

• Great resolution 

• 48 cameras with redundant coverage 

• Each camera is 5184x3456 
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Infinite-Realities Scans: 

• Great resolution 

• 48 cameras with redundant coverage 

• Each camera is 5184x3456 





Part 1: Blendshapes 


Alignment Overview 

1. Place locators at dots 

2. Wrap mesh to scan 

1. Move joints to dots 

2. Match to closest point on scan 

3. Relax 

4. Repeat 

3. Export head 

4. Project textures 

1. Iteratively apply optical flow 

• Curvature and High-Frequency Diffuse 

2. Reapply optical flow results to mesh 

5. Done 


Base Mesh 

• First task was retopo. 

• 4801 verts 

• 4684 polys (9368 tris) 

• High res, not crazy 



Joint Rig 

• About 150 joints 

• Extra joints on the neck and inner lips 




Old-School UVs 


• Flattened UVs, optimized for face. 

• Replace the middle with animated textures. 

• Animated texture has good layout and minimal stretching 

• Back of head has major stretching, but we don't care 
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Old-School UVs 

• Flattened UVs, optimized for face. 

• Replace the middle with animated textures. 

• Animated texture has good layout and minimal stretching 

• Back of head has major stretching, but we don't care 





• You can track without markers 

• HP Duiker convinced me to put some dots 
on, and I owe him big time. 

• My main regret is that I should have had 
more markers 

• Especially below the jaw! 

• And on the ridge formed by the cheeks. 



Solving: 

• Manually track markers inside Maya 

• Those green crosses are locators 

• Was done the most tedious, painful 
way possible. 

• Placed the locators entirely by hand 

• About 10-15 mins per scan. 

• Spent a few days 

• Could be done much more efficiently 



Solving: 

• Take the joint in the neutral pose, and 
find the nearest point on the rig. 

• Create rigging automatically. 

• Used in iterations. 


Mesh solving: 

• We start with the neutral mesh and want to solve it to a scan. 

• Find the locators on the scan. 

• Apply the vector translations to the mesh. 

• Call this step "SolveForLocatorsQ". 

• We are tweaking the mesh such that the locators align to the locators on the 
scan. 
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Mesh solving: 

• We start with the neutral mesh and want to solve it to a scan. 

• Find the locators on the scan. 

• Apply the vector translations to the mesh. 

• Call this step "SolveForLocatorsO". 

• We are tweaking the mesh such that the locators align to the locators on the 
scan. 



Mesh solving: 

• Another operation we can do is lock to the scan. 

• For each vertex: 

• Move the vertex to the nearest point on the scan. 

• Call this operation "LockToScanO". 
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Mesh solving: 

• Another operation we can do is lock to the scan. 

• For each vertex: 

• Move the vertex to the nearest point on the scan. 

• Call this operation "LockToScanO". 




One more operation: 

• Laplacian smoothing 

• The laplacian is the offset of a vertex from the average of its 
neighbors. 

• Intuitively, we want a solution that minimizes sharp edges. 

• Call this operation "RelaxMeshQ" 
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One more operation: 

• Laplacian smoothing 

• The laplacian is the offset of a vertex from the average of its 
neighbors. 

• Intuitively, we want a solution that minimizes sharp edges. 

• Call this operation "RelaxMeshQ" 



Solving: 

For the initial solve, we just have a big for loop. 
For a bunch of iterations: 

SolveForLocators(); 

LockToScan(); 

RelaxMeshO; 
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Solving: 

For the initial solve, we just have a big for loop. 
For a bunch of iterations: 

SolveForLocators(); 

LockToScan(); 

RelaxMesh(); 







After solving: 

• Left is both, middle is the solved rig, right is the scan 






Solving Meshes: 

• That process solves half of our problem. 

• We need to align the rig to the scan: 

• Done 

• We need the UVs to perfectly match. 

• This is the hard part. 


Project the textures as a starting point. 





Project the textures as a starting point. 




Project the textures as a starting point. 




Project the textures as a starting point. 




Project the textures as a starting point. 




Project the textures as a starting point. 





Let's take a break and talk about optical flow... 


Globally optimize optical flow. 

Inspiration: 

• Leveraging Motion Capture and 3D Scanning for High-fidelity Facial 
Performance Acquisition 

• http://facultv.cs.tamu.edu/ichai/proiects/face-TQG-2011/Face-final-vll.pdf 

• Match from optical flow using curvature 




Digital Ira 

• Lightstage 

• Optical flow with diffuse map 

• http://www. youtube. com/watch?v=lstcFOGwvU4 

• Very even lighting 



Modified approach: 

• For each scan, find the nearest scans in terms of curvature. 

• From left to right: Smile, Ss, Aa 



Modified approach: 

• In theory, all scans should converge to neutral 

• In reality, not all will converge 

• Will at least converge to "close" scans 



Curvature: 

• Export curvature for each scan. 

• Also apply some image-based noise reduction. 



Diffuse: 

• Solving diffuse does not work too well. 

• Scanning room is evenly lit, but you will never get it perfect 

• Some directionality in the lighting is built into the scans. 

• Causes optical flow to get confused. 

• Solution: Extract high frequencies from diffuse. 


Optical Flow 

• On more trick: 

• Solve for the neutral pose as well as the nearest 5 

• Median will ignore it if it does not solve well. 

• For three iterations: 

• For each pose: 

• Solve optical flow for curvature for all 5 nearest poses + neutral. 

• Take the median. 

• For each pose: 

• Solve optical flow for high-frequency of the diffuse for all 5 nearest poses + neutral. 

• Take the median 


Final Step: 

Apply optical flow to each mesh. 

Ignore optical flow around eyes, mouth, and back of the head. 


Results: 


• Before optical flow on left, after on 
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Results: 


• Before optical flow on left, after on right 





Optical Flow: 

• Effect is subtle. 

• Makes a HUGE difference during compression. 


Normals: 

• Still need a normal map. 

• Custom transfer function 

• Run image-space noise reduction 



AO: 

• We also need AO. 

• Custom AO bake 

• Use as specular mask 




Blendshapes: 

• And that is how to create blendshapes! 

• Full pipeline takes about a day or two to run. 

• Could be put on a farm. I just have one machine. 

• All maps are processed in 2k, to extract lk maps from center. 

• Have done some 8k tests, maybe later 





Blendshapes: 

• CPU Time = Cheap 

• Artist Time = Expensive 

• Very little manual work per-shape 

• About a day or two to align markers 

• Less than a day to manually tweak the lips 

• The rest is just processing 

• And, of course, writing the code 

• Fully non-destructive 

• Not an issue to rerun the whole pipeline 

• Need to rerun when the rig updates 




Step 2: Compression/Decompression 


Recap: 

• We have: 

• 70 blendshapes 

• 70 diffuse maps 

• 70 normal maps 

• 70 ao maps 

• The shapes are less of an issue. 

• The textures are a problem 


Where have I seen this problem before... 


Playable Ucap 

• GPU Gems 3: Playable Universal Capture 

• Compression was Diffuse only 

• No animated Normal or AO map 

• Entire texture was compressed 

• Includes Teeth/Tongue/Mouth Bag 

• Eyes also included 


Guide to PCA 

• Many Diffuse Textures (70) 
• Tiger Woods had thousands 







Guide to PCA 

• Actually the offset 

• Showing green channel 

• From left to right: 

• Smile, Neutral, Smile Offset (Smile 
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Guide to PCA 

• Reconstruction 

• Final = wO*imgO + wl*imgl + w2*img2 + ... + wll*imgll 

• Animate by just changing weights (shader constants) 








Guide to PCA 

• Reconstruction 

• Final = wO*imgO + wl*imgl + w2*img2 + ... + wll*imgll 

• Animate by just changing weights (shader constants) 








Calculating PCA 

• Could use SVD 

• SVD will solve all 210 columns 

• Then we chop off all but the first 12 
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Problems with SVD 

• Did this on Tiger Woods 

• Dedicated machine with lots of RAM 

• Really slow. Overnight. 

• Debugging nightmare. 

• What if it breaks? Hope it doesn't happen. 


Iterative Calculation 

• Not too difficult 

• Simple algorithm 

• Stability not an issue 

• At least not with 12 components 


PCA 


• Algorithm: From Wikipedia 

• http://en.wikipedia.org/wiki/Principal component analysis 

• Calculate first eigenvector 

• After each eigenvector, subtract out the projection of each vector into 
the eigenvector 


r — a random vector of length p 
do c times : 

g — Q (a vector of length p) 
for each row flT ^ 

s = s + (x ■ r)x 



return 1’ 



Regions 

• Calculate each eigenvector normally 



Compression Problems 

• Linear whole image 

• Hard to isolate regions 





Compression Problems 

• Stretch 

• Wants both 







Compression Problems 

• FACS 02 

• Want just the forehead, not undei 
the chin. 






Compression Problems 

• FACS 27 

• Want just the under the chin, 
the forehead. 






Corrective shapes 

• Not possible to only take part of the region 

• Could we fix this? 








PCA 

• Instead of choosing weights for each 
texture, you choose weights for each 
region 

• Adds a little bit to decompression cost 

• Greatly helps the compression 



Regions 

• Split regions into PCA components 

• Mouth left, mouth right, chin/neck 






Regions 

• Typical PCA, find scalar (lxl) that gets you as close as possible. 

• Remove, leaving residue 



Regions 

• Find optimal weights for 8 separate masked textures 

• Least squares 





Compression Problems 

• Stretch 

• Wants it 





Compression Problems 

• FACS 02 

• Want just the forehead, not under 
the chin. 







Compression Problems 

• FACS 27 

• Want just the under the chin, not 
the forehead. 





Regions 

• By isolating which region get which PCA components, solve is much 
better 

• Especially for first few components 






Results with 12 components 
Left is before, right is after. 
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Results with 12 components 
Left is before, right is after. 






PCA 

Definitely loses some detail. 

Could be much, much worse. 

When there is texture swimming, it loses detail fast. 





Shader: 


flcat4 regionG = g_txPcaRegion0, Sample( g_samLinearClamp > input ,TextureUV ) , xyzw j 
float4 regionl = g_txPcaRegionl. Sample( g_samLinearClamp J input ,TextureUV ) , xyzwj 

float regionData[8] = { region0,x., region0,yj region0,Zj. regionG, regionl, x, regionl, y., regionl, Zj. regionl, w 


[unroll] for (iter = iter < Bi iter44) 

{ 

float weight = regionData[iter] j 
diffRG += weight * g_pcaMaskDiff RG[iter] j 
diffRl 4= weight * g_pcaMaskDiff Rl[iter] j 
diffR2 4= weight * g_pcaMaskDiff R2[iter]^ 


diffGG 4= weight * g_pcaMaskDiffGG[iter] j 
diffGl 4= weight * g_pcaMaskDiffGl[iter]^ 
diffG2 4= weight * g_pcaMaskDiffG2[iter] j 


diffB0 4= weight * g_pcaMaskDiff BG[iter]i 
diffBl 4= weight * g_pcaMaskDiff Bl[iter]^ 
diffB2 4= weight * g_pcaMaskDiff B2[iter] j 


flcat3 baseDiff = g_txDiff use, Sample ( g_samLinearClamp > input, TextureUV ) , rgbi 


float4 comp0 = g_txPcaDiffG, Sample ( g_samilinearClamp. f pcaUv ).rgba * 2,0f - l,Gf; 
float4 compl = g_txPcaDiffl, Sample ( g_samilinearClampj. pcaUv ),rgba * 2,0f - l,Gf; 
float4 comp2 = g_txPcaDiff 2, Sample( g_samilinearClampj pcaUv ),rgba * 2,0f - l.&fj 


ret,x = baseDiff, x 4 dot (diff RG^ com pG) 4 dot (diffRl^ compl) 4 dot (diff R2_ h comp2)j; 
ret,y = baseDiff, y 4 dot ( diff GG^compG) 4 dot (diffGl, compl) 4 dot (diffG2 > comp2) j 
ret,z = baseDiff, z 4 dot (diff BG.* com pG) 4 dot (diff Bl, compl) 4 dot (diff B2 > comp2)j 


ret = pow( saturate (ret) j2, 2f )^ 




Old-School UVs 

• Flattened UVs, optimized for face. 

• Replace the middle with animated textures. 

• Animated texture has good layout and minimal stretching 

• Back of head has major stretching, but we don't care 





PCA Notes 

• Use zoomed neutral expression as mean 

• 70 diffuse maps as 4, lk maps 

• 1 regular and 3 zoomed 

• Much cheaper than 70, 2k maps! 

• Same memory as a single 2k texture 







PCA Notes 

• Normal/AO uses same method, zoom maps are 512 

• Actual data is 512 

• You could do larger 

• AO base is stored in alpha of Normal, but has separate PCA tex 







Total Diffuse/Normal/AO Texture Memory 

• Diffuse, Normals, AO, + PCA for all 3 

• Plus tiny regions weights 

• Everything BC7 compressed 

• Custom encoder 

• Total, 8.7 MB 

• Not including teeth/tongue/eyes/stretch discussed later 







PCA Notes 

• Could be used for standard wrinkle normal maps 

• XYZ have good coherency 

• Single 2k map is 5.33MB with BC7 

• Neutral, Compress, Stretch 





Could be useful for other things 

• Easy to blend for LOD 

• Scale weights to 0 

• Easy to scale quality 

• 12 components is arbitrary 

• Easy decompression 

• Free sampling, mipmapping, etc. 

• Clint Hanson had water example in 2006 

• Baked lightmaps for time of day? 

• Bake out every time of day and compress with PCA? 


Hooray, blending! 





Hooray, blending! 



i 



Hooray, blending! 





Part 3: Driving with Mocap 


So we have blendshapes... 

• FACS 24, FACS 33, Compress, Smile, Surprise 



And we have mocap... 
• Now what? 





So we have blendshapes... 

• Blendshapes defined by vertices 

• Animation defined by joints 




So we have blendshapes... 

• Define blendshapes by joints 





Problem: Correspond Blendshapes 
• Find weights for shapes that match joint animation. 






Problem: Correspond Blendshapes 
• Use those weights for blendshapes. 





Let's say we have a joint. 

• Current frame animation has offset from neutral 

• Find set of shapes to match that offset 

• Let's say we are on the left eyebrow. 


Case 1: 

• A 0 = [0,5,0] 
•A 1 = 

• B = 

• A 0 x + A x y = B 

• x=?, y=? 
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Case 1: 

• Aq = [0,5,0] 

• A j = [1,5,0] 
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• A 0 x + Ajy = B 

• x=?, y=? 



Case 1: 

• Aq = [0,5,0] 

• A j = [1,5,0] 

• B = [1,10,0] 

• A 0 x + Ajy = B 

• x=l, y=l 




Case 2: 
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Case 2: 

• A 0 = [0,5,0] 
•A 1 = 

• B = 

• A 0 x + A x y = B 

• x=?, y=? 




Case 2: 

• Aq = [0,5,0] 

• A t = [1,5,0] 

• B = 

• A 0 x + Ajy = 

• x=? / y=? 



Case 2: 

• Aq = [0,5,0] 

• A t = [1,5,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 

• x=?, y=? 




Case 2: 

• Aq = [0,5,0] 

• A t = [1,5,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 

• x=-l, y=l 




Case 2: 

• A 0 = [0,5,0] 

• A 1 = [1,5,0] 

• B = [1,0,0] 

• A 0 x + A x y = B 

• x=-l, y=l 



"If at first you don't succeed, try, try 
again. Then quit. There's no point being 
a damn fool about it." 


- W. C. Fields 



Solution: 

• Our matrix is overconstrained 

• Disallow negative weights 

• Can't do a "negative" eyebrow up 

• Use Non-Negative Least Squares 


Case 2: 

• Aq = [0,5,0] 

• A t = [1,5,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 

• x=0, y=0 
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Case 3: 


Ai = 

a 2 = 

B = 

A 0 x + A x y + A 2 z= B 
x=?, y=?, z=? 
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Case 3: 

• A 0 = [0,5,0] 

• A 1 = [1,5,0] 

•a 2 = 

• B = 

• A 0 x + A x y + A 2 z= B 

• x=?, y=?, z=? 




Case 3: 

• Aq = [0,5,0] 

• A t = [1,5,0] 

•a 2 = 

• B = [1,0,0] 

• A 0 x + Ajy + A 2 z= B 

• x=?, y=?, z=? 




Case 3: 

• Aq = [0,5,0] 

• A, = [1,5,0] 

• A 2 = [.1,0,0] 

• B = [1,0,0] 

• A 0 x + Ajy + A 2 z= B 

• x=?, y=?, z=? 




Case 3: 

• Aq = [0,5,0] 

• A, = [1,5,0] 

• A 2 = [.1,0,0] 

• B = [1,0,0] 

• A 0 x + Ajy + A 2 z= B 

• x=0, y=0, z-10 




Solution: 

• Disallow weights greater than 1.0 

• Might go a little higher in some cases (like 1.5) 

• Could use a proper range solver 

• I just clamped at the end 

• Somewhat rare case 


Case 3: 

• Aq = [0,5,0] 

• A, = [1,5,0] 

• A 2 = [.1,0,0] 

• B = [1,0,0] 

• A 0 x + Ajy + A 2 z= B 

• x=0, y=0, z-10 




Case 3: 


• A 0 = [0,5,0] 

• A 2 = [1,5,0] 

• A 2 = [.1,0,0] 

• B = [1,0,0] 

• A 0 x + A 2 y + A 2 z= B 

• x=0, y=0, z-1 
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Case 4: 


Ai = 

B = 

A 0 x + A x y 
x=?, y=? 


Case 4: 

• A 0 = [0,5,0] 
•A 1 = 

• B = 

• A 0 x + Ajy = B 

• x=?, y=? 
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• A t = [l,-5,0] 

• B = 
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• A t = [l,-5,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 
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Case 4: 

• A 0 = [0,5,0] 

• A t = [l,-5,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 

• x=l, y=l 





Usual Solution: 

• Disallow certain blendshape combinations 

• Can't have brow up and brow down 

• Lot's of combinations to worry about 

• Did not do this. 


Solution: 

• Change the joints data 

• Instead of 3 vector [x,y,z] 

• Make it 6 vector [+x,-x,+y,-y,+z,-z] 


Case 4: 

• A 0 = [0,5,0] 

• A t = [l,-5,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 

• x=l, y=l 





Case 4: 

• A 0 = [0,5,0] -> [0,0,5,0,0,0] 

• A t = [l,-5,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 

• x=l, y=l 





Case 4: 

• A 0 = [0,5,0] -> [0,0,5,0,0,0] 

• A t = [l,-5,0] -> [1,0,0,5,0,0] 

• B = [1,0,0] 

• A 0 x + Ajy = B 

• x=l, y=l 





Case 4: 

• A 0 = [0,5,0] -> [0,0, 5, 0,0,0] 

• A, = [1,-5, 0] -> [1,0,0,5,0,0] 

• B = [1,0,0] -> [1,0, 0,0, 0,0] 

• A 0 x + A^ = B 

• x=l, y=l 





Case 4: 

• A 0 = [0,5,0] -> [0,0, 5, 0,0,0] 

• A, = [1,-5, 0] -> [1,0,0,5,0,0] 

• B = [1,0,0] -> [1,0, 0,0, 0,0] 

• A 0 x + A^ = B 

• x=l, y=l -> [1,0, 5, 5,0,0] 





Case 4: 


• A 0 = [0,5,0] -> [0,0, 5, 0,0,0] 

• A, = [1,-5, 0] -> [1,0,0,5,0,0] 

• B = [1,0,0] -> [1,0, 0,0, 0,0] 

• A 0 x + A^ = B 

• x=l, y=l -> [1,0, 5, 5,0,0] 

• x=0, y=0 -> [0,0, 0,0, 0,0] 





System setup: 

• Each shape has 150 joints 

• 6 channels per joint [+x,-x,+y,-y,+z,-z] 

• 900 channel vector 

• Matrix is 900 rows with 70 columns 

• Solve for 70 weights 


Other pieces: 

• Results are sparse 

• Ignore blendshapes for eyelids and lips 

• Just use joint animation 

• Do this for each region separately (8 times) 


Part 4: Rendering 




Quick Recap 

• After all that work, what we have is 

• Triangle Mesh 

• Diffuse Map 

• Normal Map 

• AO Map 

• How to render? 


Quick Overview: 

• Skin SSS 

• AO with spherical harmonics 

• Adaptive Tessellation 

• Eyes 

• Teeth 


Skin SSS 


• Remember the NVIDIA Doug Jones Head? 

• http://http.developer.nvidia.com/GPUGems3/gpugems3 chl4.html 

• "Bible" of skin shading 

• Goal is to make it cheaper while retaining quality. 


Skin Shader: Texture Space Blur 

• How to actually do the blur? 

• In theory, 5 gaussians 

• Separable, but still many passes 
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Skin Shader: Texture Space Blur 

• Discussion with Morten Mikkelson 

• We both agreed: It's a blur and a spike 

• Exact shape of blur doesn't matter 

• Same conclusion in Ryse talk two days ago by Nicolas Schulz. 


Reflectance 
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Skin Shader: Texture Space Blur 

1. Render to lightmap 

2. Blur vertically 

3. Blur horizontally 

• Combine all 5 blurs into one 

• It's wrong. Yet, I'm still able to sleep at 
night. 

• How big is the map? 





How Big? Rendering at 1080p. 


Answer: 256x256 


Answer: 256x256 



Texture Space 

• Less publicized advantage of texture space. 

• Much lower texel density than the screen space size. 

• Because it's a BLUR! It doesn't need high frequency information. 

• Texture space better in some cases, Screen Space better in others. 


Skin Shader: One pass approximation 
• We can also do it in one pass. 




Skin Shader: One pass approximation. 
• Top row with detail normal 



Variance 
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Skin Shader: One pass approximation. 
• Top row with detail normal 



Variance 



0.0064 0.233 0.455 0.649 

0.0484 0.100 0.336 0.344 

0.187 0.118 0.198 0 

0.567 0.113 0.007 0.007 

1.99 0.358 0.004 0 

7.41 0.078 0 0 




Skin Shader: One pass approximation. 
• Blur accumulates nearby pixels 




Variance 

(mm A 2) 

Red 

Blur Weights 
Green 

Blue 
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Skin Shader: One pass approximation. 

• Assume that normals cancel out 

• Approximate all samples with geometry 
normal 



Variance 

(mm A 2) 

Red 

Blur Weights 
Green 

Blue I 
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Skin Shader: One pass approximation. 

• Top row lighting with detail normal 

• Everything else with geometry normal 
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Skin Shader: One pass approximation 

• N = normal mapped normal 

• G = geometry normal 

• DiffN = DiffuseLight(N) 

• DiffG = DiffuseLight(G) 

• ColorRatio = (.233,-455,. 649) 

• Diff = lerp(DiffN, DiffG, ColorRatio) 




Comparison 

• Texture Blur 
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Comparison 

• Geometry/Detail Blend 



Comparison 
• Lambert 



Comparison 
• Texture Blur 
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Comparison 


• Geometry/Detail Blend 




Comparison 
• Lambert 
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Comparison 

• Geometry/Detail Blend 



Comparison 
• Lambert 



Things that affect need for SSS 

• Harsh lighting needs more SSS than flat lighting 

• Farther characters don't need SSS 

• SSS is only a few mm anyways 

• Less than one pixel far away 


Powdered doughnuts: 




Powdered doughnut problem: 

• Evan Wells came up to me and said 


Powdered doughnut problem: 

• Evan Wells came up to me and said (paraphrasing) 


"There's a big problem with the 
cutscene! Drake looks like he ate a 
powdered doughnut!!" 


Powder Doughnut 




Solution: Shadow 




Shadows 

• Easy! Just add a shadow. 

• We have no limit on those, right? 

• Better option? 


Spherical Harmonic Ambient Occlusion 

• It's like ambient occlusion 

• But with spherical harmonics! 

• Bake SH per-vertex 

• Figures out where light can and can not come from. 


SHAO: No shadow or SHAO 




SHAO: SHAO 







SHAO: With shadow 




SHAO: No shadow or SHAO 




SHAO: SHAO 




SHAO: With shadow 




SHAO 

• Full rendering 

• 4 lights, only one with a shadow 


SHAO: No shadow or SHAO 



SHAO: SHAO 



SHAO: With shadow 




SHAO: With shadow and SHAO 




Powder doughnut: 

• Unshadowed lights causing highlights 

• Can also see it in the nostrils, and ears 

• Made worse by proper physically based shading 

• Bright rimlights 


SHAO 


• Not as good as a shadow. 

• Way better than nothing. 

• Too expensive to use everywhere (vertex cost) 

• A major help on unshadowed lights affecting the head. 

• Easy to use with forward rendering 

• Probably too hard deferred 


Adaptive Tessellation 

• We can have tessellation this generation. 

• Yes, we say that every generation. 

• But now I mean it! 

• Tessellating the whole thing is too expensive in all but extreme cases. 

• But can we tessellate only the silhouette 


Tessellation Off 



Tessellation Adaptive 



Tessellation Everywhere 



Tessellation Off 





Tessellation Adaptive 





Tessellation Everywhere 





Tessellation Adaptive 

• Times on NVIDIA 660ti 

• Off: 4.14ms 

• Adaptive: 4.22ms 

• Everywhere: 6.18ms 


Adaptive Tessellation 

• DX11 is awesome 

• Hull shader with fractional_odd partitioning 

• Each vertex calculates "tessellation weight" 

• Edge tessellation commutative function of both verts 




Adaptive Tessellation 

• DX11 is awesome 

• Hull shader with fractional_odd partitioning 

• Each vertex calculates "tessellation weight" 

• Edge tessellation commutative function of both verts 





Adaptive Tessellation 

• DX11 is awesome 

• Hull shader with fractional_odd partitioning 

• Each vertex calculates "tessellation weight" 

• Edge tessellation commutative function of both verts 




Adaptive Tessellation 
• Const hull shader 


□ HS_CONSTANT_DATA_TRI PhongTessConstHS ( Input Pat ch<VS_OUTPUT_HEAD, 3> input ) 
|{ " 

H S_CON STANT_D ATA_T R I outputj 

float tess® = input [G] .TessValj 
float tessl = input [1] .TessValj 
float tess2 = input [2] .TessValj 

output .Weights = float3(tess€jtessljtess2) j 

float contrast = 1.2j 

output .MaxVal = saturate(contrast*( (tessfi 4 tessl 4 tess2)/3. , 0f ) ) j 

output . Edges[G] = .5*(tessl4tess2)*g_tessFactorEdge)4l.@; 

output . Edges[l] = maKfe^ .5*(tess24tess0)*g_tessFactorEdge)4l. , 0; 
output . Edges[2] = maxfGj .5*(tess , 04tessl) I|= g_tessFactorEdge)4l.'0j 
output . Interior[0] = output .MaKVal*g_tessFactorEdgej 

return outputj 

b 



Adaptive Tessellation 
• Hull shader 


[ domain ("tri” ) ] 

[ partitioning (”f ractional_odd”)] 

[outputtopology ( ll triangle_cw" ) ] 

[outputcontrolpoints(3)] 

[patchconstantf uncf^PhongTessConstHS 11 ) ] 

VS_GUTPUT_HEAD PhongTessH5( Input Pa tch<VS_0UTPUT_HEAD,3> patchy 

uint index : SV_OutputControlPointID) 


} 


return patch[index]i 



Adaptive Tessellation 

• Domain shader 

• Phong Tesselation 


[ domain ("tri”)] 

VS_OUTPUT_HEAD PhongTessDS ( HS_CGNSTANT_DATA_TRI inputs 

float3 uv : SV_DomainLocation J 

const Output Patch < VS_OUTPUT_H E AD ^ 3> patch) 

{ 

VS_OUT P UT_H E AD ret = (VS_OUTPUT_HEAD )0j 

AccumulatVertexfretjUv.x., patch [0]); 

AccumulatVertex( ret^ uv, patch [1] ) j; 

AccumulatVertexfret^uv, z^patch[2])^ 

float3 wpBase = ret , HorldPos; 
floats normBase = ret , Normal; 

float 3 wp® = Pro jectPoint (patch [0] .WorldPoSj patch [0] .NormaljWpBase); 
float3 wpl = Pro jectPoint (patch[l] -WtorldPoSj patch [1] .INormaljWpBase); 
float3 wp2 = ProjectPolnt (patch[2] -HorldPos J patch[2] .NormaljWpBase); 

float len0 = DistLinePlane(patch[0] » WorldPo-s.* patch [0] , NormaljWpBase., normBase) ; 
float lenl = DistLinePlane(patch[l] „ WorldPos^patchfl] , NormaljVNpBase^normBase); 
float len2 = DistLinePlane( patch[2] . WorldPos^patch[2] , No rma 1 j wp B a se^ normBase); 

float alpha = input -Weight s.x*uv.x + input. Weights. y*uv,y + input , Weights, z*uv, z; 
float tessFactorAlpha = alpha * g_tessFactorAlpha; 

float3 wpDst = tessFactorAlpha* (uv.x*wp0 + uv.y*wpl + uv,z*wp2) + (1.0f -tessFactorAlpha) *wpBase; 
float4 projPos = mul( g_mWorldViewPro float4(wpDst J l-0) ); 
ret, Position = projPos; 
return ret; 



Eyes: Shader 



Eyes: Big thing 
• AO 


Eyes: Big thing 

• AO baked into Lat-Long space. 


Eyes: Shader 





Eyes: Shader 

• Compress all 70 maps with PCA 

• Drive using Y of eye joints 





Teeth: Big thing 
• AO 


Teeth: Big thing 

• AO based on light source in front 
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Teeth: AO 




Teeth: AO 





Teeth: AO 



Teeth: AO 

• Compress all 70 maps with PCA 

• Drive with Y of mouth joints 


Conclusions: 

• Overall, it works 

• Diffuse animation is essential 

• Animation doesn't always use the best blendshapes 

• Wrinkles under the eyes 

• Would be better solved to a proper rig 

• Real games need ways to adjust the meshes 

• Art director wants a smaller nose, don't want to resculpt 70 expressions 


Credits: 

• Infinite-Realities (3d scans) 

• Ramahan Faulk (character modeling) 

• Mocap Militia (mocap shoot) 

• Matthew Mercer (mocap talent) 

• Special Thanks: HP Duiker, Maggie Bellomy, Habib Zargarpour 


Questions? 


